Increasing frame rate for imaging

ABSTRACT

The specification and drawings present a new method, apparatus and software product for implementing algorithms for improving a frame rate by reducing a size and resolution of selected versus unselected image frames captured by a sensor in an image processing pipeline before displaying (or before encoding) in electronic devices (e.g., cameras, wireless camera phones, etc.), according to various embodiments of the present invention.

TECHNICAL FIELD

The present invention relates generally to image processing in camerasand electronic devices and, more specifically, to improving frame rateby reducing a size and a resolution of selected video image frames in animage processing pipeline before displaying or encoding.

BACKGROUND ART

Image processing in current digital camera systems is done either byusing a hardware (HW) implementation, a software (SW) implementation ortheir combination thereof. The HW implementation can be integrated to acamera head, a system IC (integrated circuit) or be a separatecomponent. The HW implementation is typically fast and can achieve highframe rates for real time operations like viewfinder and video. Thedownside of the HW implementation is the added HW cost and inflexibilityin regard to image quality tuning needs.

On the other hand the SW implementation of the image processing pipelinemay be cost-effective and flexible, and may be tuned easily. However, atypical constraint in the SW based implementation is availableprocessing power or MIPS (million instructions per second, a processorperformance measure). Real time operations like video recording anddisplaying the image on a viewfinder in the SW based image processingpipeline may be especially problematic. This is because there is a fixedtime budget and all operations including data transfer from camera(e.g., from an image sensor) to a display have to be done in givenamount of time. For example, in order to achieve a frame rate of 15frames per second (fps), only 66.7 milliseconds (ms) is available toperform all operations, and to achieve 30 fps only 33 ms is available,etc. To achieve such a fast processing may be problematic in low costprocessor systems (especially for low-light video applications), e.g.,with desired higher picture/screen resolutions.

This problem with SW implementation may be partially reduced byoptimizing the image processing SW, i.e., by dropping certain imageprocessing operations, e.g., like lens shading correction. Optimizationsmay be possible to a certain extent but there is a limit to what degreethe processing time can be reduced because dropping image processingoperations will degrade the image quality.

Another approach may be using downscaling/upscaling for all frames inthe image. In this method a lot of image information may be lost and theimage quality may be noticeably degraded.

DISCLOSURE OF THE INVENTION

According to a first aspect of the invention, a method may comprise:processing a plurality of image frames of an image, wherein theprocessing comprises reducing a size of selected image frames chosenfrom the plurality of image frames relative to an original or a reducedsize of unselected image frames of the plurality of image frames,wherein a processing time for the selected image frames is shorter thanfor the unselected image frames; and correcting the selected imageframes after the processing for displaying the image with a targetedframe rate and a desired quality level using interleaving of theselected and unselected image frames after the correcting or after thecorrecting followed by further processing of the selected and unselectedimage frames.

According further to the first aspect of the invention, the reducing ofthe size of the selected image frames may be proportional to reducing aresolution of the selected image frames, and, if the original size ofthe unselected image frames is reduced, the reduced size of theunselected image frames is reduced in the same proportion as aresolution of the unselected image frames.

Further according to the first aspect of the invention, the correctingmay comprise: scaling up the selected image frames to be equal in sizeto the unselected image frames using a pre-selected method; and delayingthe selected image frames by a predetermined time interval relative tothe unselected image frames to maintain a constant frame rate to providethe displaying of the image with the targeted frame rate and the desiredquality level.

Still further according to the first aspect of the invention, thecorrecting after the processing may comprise scaling up the selectedimage frames to be equal in size to the unselected image frames using apre-selected method and the further processing comprises encoding of theselected and unselected image frames with subsequent storing or sendingthe selected and unselected frames to a desired destination for thedisplaying of the image with the targeted frame rate and the desiredquality level.

According yet further to the first aspect of the invention, theprocessing may comprise processing into proper color images whichcomprises at least one of: a Red-Green-Blue color space format or acolor space format having three color components, wherein one colorcomponent is a luminance color component and two other color componentsare chrominance color components.

According still further to the first aspect of the invention, a numberof the selected image frames and a number of the unselected image framesmay be determined automatically or inputted through a user interface.

According further still to the first aspect of the invention, a numberof the selected image frames and a number of the unselected image framesmay automatically vary depending on a moving speed of one or moreobjects in the image in order to provide the targeted frame rate and thedesired quality level.

According to a second aspect of the invention, a computer programproduct may comprise: a computer readable medium embodying a computerprogram code thereon for execution by a computer processor with thecomputer program code, wherein the computer program code comprisesinstructions for performing the first aspect of the invention.

According to a third aspect of the invention, an apparatus may comprise:an image signal processor, configured to perform processing of aplurality of image frames of an image, the processing comprises reducinga size of selected image frames chosen from the plurality of imageframes relative to an original or a reduced size unselected image framesof the plurality of image frames, wherein a processing time for theselected image frames is shorter than for the unselected image frames;and a frame synchronization controller, configured to provide aselection signal to choose the selected image frames, wherein theapparatus is configured to provide correcting the selected image framesafter the processing in order to display the image with a targeted framerate and a desired quality level using interleaving of the selected andunselected image frames after the correcting or after the correctingfollowed by further processing of the selected and unselected imageframes.

Still yet further according to the first aspect of the invention, theapparatus may further comprise: a display image scalar, configured toscale up, after performing the processing, the selected image frames tobe equal in size to the unselected image frames using a pre-selectedmethod as a part of the correcting.

Further according to the third aspect of the invention, the apparatusmay further comprise: a delay unit, configured to delay the selectedimage frames, after performing the processing, by a predetermined timeinterval relative to the unselected image frames to maintain a constantframe rate as a part of the correcting to provide the displaying of theimage.

Still further according to the third aspect of the invention, theapparatus may further comprise: an encoder, configured to perform thefurther processing by encoding the selected and unselected image frameswhich is followed by storing or sending the selected and unselectedframes to a desired destination for the displaying of the image.

According yet further to the third aspect of the invention, theplurality of image frames may be raw image data and the apparatus maycomprise a sensor configured to provide the raw image data signal. Stillfurther, the sensor may be a complimentary metal oxide semiconductor ora charged-couple device.

According still further to the third aspect of the invention, the imagesignal processor and the frame synchronization controller may becombined in one unit or a processor.

According yet further still to the third aspect of the invention, acamera or an electronic device comprising a camera may comprise theimage signal processor and the frame synchronization controller.

Still yet further according to the third aspect of the invention,wherein, during the processing, the selected image frames may be reducedto a size of 160×120 pixels, and the unselected image frames of theplurality of image frames may be reduced to a size of 320×240 pixels.

According further still to the third aspect of the invention, each ofthe selected image frames may be chosen to be every second frame fromthe plurality of image frames.

Yet still further according to the third aspect of the invention, theapparatus may be configured to determine using a processor performancemeasure of the image signal processor in order to achieve the targetedframe rate and the desired quality level at least one of: a number ofthe selected image frames and a number of the unselected image frames,and a size of the selected image frames and a size of the unselectedimage frames.

According to a fourth aspect of the invention, a processor may comprise:an image signal processor, configured to perform processing of aplurality of image frames of an image, the processing comprises reducinga size of selected image frames chosen from the plurality of imageframes relative to an original or a reduced size of unselected imageframes of the plurality of image frames, wherein a processing time forthe selected image frames is shorter than for the unselected imageframes; and a frame synchronization controller, configured to provide aselection signal to choose the selected image frames, wherein theapparatus is configured to provide correcting the selected image framesafter the processing in order to display the image with a targeted framerate and a desired quality level using interleaving of the selected andunselected image frames after the correcting or after the correctingfollowed by further processing of the selected and unselected imageframes.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the nature and objects of the presentinvention, reference is made to the following detailed description takenin conjunction with the following drawings, in which:

FIG. 1 is a flow chart for implementing an algorithm for improving aframe rate by reducing a size/resolution of selected image frames in animage processing pipeline before displaying (or before an encodingstage), according to an embodiment of the present invention;

FIGS. 2 a-2 c are timing diagrams demonstrating alternating one fullsize unselected image frame (or one reference size frame correspondingto a user requested video image size) with one reduced size selectedimage frame, according to an embodiment of the present invention;

FIGS. 3 a-3 c are timing diagrams demonstrating alternating one fullsize unselected image frame (or one reference size frame correspondingto a user requested video image size) with three reduced size selectedimage frames, according to an embodiment of the present invention; and

FIG. 4 is a block diagram of an electronic device (e.g., a camera, acamera phone, etc.) for improving a frame rate by reducing asize/resolution of selected image frames in an image processing pipelinebefore displaying (or before encoding stage), according to an embodimentof the present invention.

MODES FOR CARRYING OUT THE INVENTION

A new method, apparatus and software product are presented forimplementing algorithms for improving (i.e., increasing) a frame rate byreducing a size and resolution (i.e., size/resolution) of selected (orchosen) versus unselected image frames captured by a sensor in an imageprocessing pipeline before displaying (or before encoding stage) inelectronic devices (e.g., cameras, wireless camera phones, etc.),according to various embodiments of the present invention.

Indeed, the amount of processed pixels by an image signal processor(ISP) in a SW (software) imaging pipeline (which, for example, may beprocessing the captured raw image data into proper color images) may bereduced by decreasing the size and resolution of the output for theselected frames by selecting signals from certain pixels and discardingothers (e.g., by selecting signals of only every second (third or forth,etc.) row or column of pixels in these selected frames). This may reducea processing time significantly in the ISP thus reducing a processorload. After processing, the selected reduced frames may be correctedbefore displaying (e.g., real time displaying on a viewfinder), orfurther processing. For example, the selected reduced frames may bescaled up to be equal in size to a reference size of unselected frames(i.e., a full or reduced size of unselected frames, i.e., reduced to auser requested video image size) using a fast upscaling operation asknown in the art. Also, in order to avoid jumpy video image displayed ona display/viewfinder, the selected frames may be further corrected bybuffering, i.e., delaying, in order to get the frame timing to match inreal time by maintaining a constant display frame rate (as illustratedin FIGS. 2 b-2 c and 3 b-3 c described herein).

It is further noted that the unselected image frames may have theoriginal full size (e.g., corresponding to the total number of pixels ofthe sensor) or the reduced size by selecting signals from certain pixelsand discarding others (e.g., by selecting signals of every other row orcolumn of pixels in these unselected frames). But even when theunselected image frames are reduced, their size should be larger (i.e.,comprising more pixels) than the selected image frames. Using unselectedframes with the reduced size may further reduce the processing time inthe ISP.

According to an embodiment of the present invention, in order to keep agood image quality, alternate unselected image frames (also known asbase frames or intra-frames) having the reference size and selectedimage frames (also known as inter-frames or delta frames) having thereduced size may be interleaved. This interleaving may provide a higherframe rate without significant loss of the video image quality. Thequality level of the image may be dependent on the targeted frame rateand the number of selected frames between the unselected frames, as wellas on the quality (e.g., related to the resolution determined by anumber of original pixels left in those frames) of the selected imageframes and/or of the unselected image frames.

Generally, the number of the selected image frames may be chosen to belarger than, to be equal to or to be smaller than the number ofinterleaving unselected image frames. For example, an alternatingfrequency of this interleaving may correspond to having, e.g., everysecond reference size (e.g., full size) image frame, every thirdreference size (e.g., full size) image frame, etc. Another scenario maybe, e.g., having an even number (e.g., 2) of the unselected image framesfollowed by a larger uneven number (e.g., 3) of the selected imageframes, or an uneven number (e.g., 3) of the unselected image framesfollowed by a smaller even number (e.g., 2) of the selected imageframes. Alternatively, the number of unselected image frames may bechosen to be larger than the number of the selected image frames, e.g.,two unselected image frames may be followed by one selected image frame,or three unselected image frames may be followed by one or two selectedimage frames, etc.

The trade-off decision for choosing the alternating number of theselected and unselected image frames chosen for this interleaving maydepend on a targeted frame rate, on a desired quality level of the videoimage, on the available MIPS (million instructions per second, processorperformance measure) of the ISP since the SW-ISP computational powerrequirement may be directly proportional to the output image size, e.g.,to the number of pixels in the output image (a smaller image will takeless time to process than a bigger image), etc.

According to another embodiment, the size (e.g., a number of pixels intwo perpendicular directions of the sensor) of the selected frames andthe size of the unselected frames (having, e.g., an original sizecaptured by the sensor, or a reduced size) may be chosen in a broadrange based on a system request/requirements, a user request (provided,e.g., through a user interface), the total number of pixels in thesensor capturing the image, the available MIPS of the ISP and/or desiredimage quality level (wherein the size of the selected frames is smallercompared to the size of the unselected frames, as described herein).

Moreover, according to a further embodiment, since the frame rate andimage quality are two desirable parameters, then the number of theselected and unselected image frames and/or their sizes (i.e., thenumber of pixels) may be chosen, e.g., based on the available MIPS ofthe ISP.

Moreover, the number of selected vs. unselected image frames may bedetermined automatically by the electronic device (or the electronicdevice comprising camera) using available processing power (e.g.,measured by the MIPS) of the ISP or may be inputted by a user through auser interface (UI). Furthermore, the number of selected versusunselected frames may be variable (i.e., their ratio may be a variableparameter) and chosen automatically by the electronic device (e.g., by acamera control system software) depending, for instance, on a movingspeed of one or more objects in the image in order to provide, e.g., thetargeted frame rate and the desired quality level.

According to another embodiment, improving the frame rate by reducingthe size/resolution of the chosen selected frames in the imageprocessing pipeline may be further used for further processing, e.g.,for encoding stage with subsequent storing the image data (which may bedisplayed later if requested) or sending the encoded image data,processed according to embodiments of the present invention, to adesired destination.

Thus, according to an embodiment of the present invention, a timerequired for processing a plurality of image frames (or video imageframes) of an image (e.g., provided by a sensor as a raw image datasignal) may be decreased by reducing the size/resolution of the selectedimage frames chosen from said plurality of image frames relative to anoriginal or a reduced size/resolution of unselected image frames of saidplurality of image frames, thus reducing the processing time for theselected image frames compared with the processing time required for theunselected image frames. Then after correcting the selected image framesfollowing this processing (the correcting may include a time correctionand upscaling) this image with the targeted frame rate and the desiredquality level may be displayed (e.g., on a viewfinder or a touch screen)using interleaving of the corrected selected image frames and unselectedimage frames. Also after this correction, a further processing of theselected and unselected image frames may be performed (e.g., encodingwith subsequent storing or sending these selected and unselected framesto a desired destination).

FIG. 1 shows a flow chart for implementing an algorithm for improving aframe rate by reducing a size/resolution of selected image frames in animage processing pipeline before displaying (or before encoding stage),according to an embodiment of the present invention.

The flow chart of FIG. 1 only represents one possible scenario amongothers. It is noted that the order of steps shown in FIG. 1 is notabsolutely required, so in principle, the various steps may be performedout of order. In a method according to the embodiment of the presentinvention, in a first step 10, a video image may be captured using asensor, e.g., such as a CMOS (complimentary metal oxide semiconductor),a CCD (charge-coupled device) or other applicable sensors, of a cameraor an electronic device (e.g., a camera-phone for wirelesscommunications) to provide a plurality of image frames, e.g., a rawimage data signal in a Bayer image format, to an image signal processor(ISP) or a software ISP (SW-ISP). The resolution of the output raw imagemay be slightly bigger than the user requested video image size.

In a next step 12, the plurality of captured frames (e.g., raw imagedata signal) may be processed by the ISP using a shorter processing timefor selected frames (known as inter-frames or delta frames) vs.unselected (known as base frames or intra-frames) frames by reducing thesize and resolution (size/resolution) of the selected image framesrelative to unselected image frames (or referenced size frames),according to an embodiment of the present invention described herein.

Generally, the size (e.g., a number of pixels in two perpendiculardirections of the sensor) of the selected frames and the size of theunselected frames (having, e.g., an original size captured by thesensor, or a reduced size) may be chosen in a broad range based on asystem request/requirements, a user request (provided, e.g., through auser interface), the total number of pixels in the sensor capturing theimage, the available MIPS of the ISP and a desired image quality level,as described herein.

For example, the unselected reference size frames provided by the CMOSsensor may be processed in step 12 in such a way that their size may be,reduced to a size of a QVGA (quarter video graphic array) image framethus comprising 320×240 pixels, whereas the selected image frames may bereduced to a size of QQVGA (quarter QVGA) image frame, having 160×120pixels. Since reduction in the image frame size (e.g., by selectingcertain rows and columns and discarding others) may generally beproportional to the reduction in the resolution, then the term “reducingsize/resolution” means reducing both size and resolution by the sameproportional value in the context of various embodiments of the presentinvention.

The output image frame resolution and size provided by the ISP may bedependent upon the instruction provided by a frame synchronizationcontroller (FSC, e.g., shown as module 54 in FIG. 4) using anappropriate command (as shown in FIG. 4), and the reduced image framesignal corresponding to the elected image frames may have a shorterprocessing time then for the unselected image frames as demonstrated inexamples of time diagrams of the output image frames signals generatedin step 12 as shown in FIGS. 3 b and 3 b and further described herein.

It is also noted that the number of unselected vs. selected image frames(e.g., per second) may vary in a broad range based on the targeted framerate, the desired quality level and the available MIPS of the ISP, andthe number of the selected image frames may be chosen to be larger than,to be equal to or to be smaller than the number of the unselected imageframes as described herein. Also, according to another embodiment, thenumber of the selected vs. unselected image frames may be determinedautomatically by the camera/device or inputted by a user through the UI(e.g., by providing a UI input to the FSC 54 as shown in FIG. 4), or maybe a variable parameter chosen automatically by a camera/devicedepending, for example, on a moving speed of an image (object) in orderto balance, e.g., the targeted frame rate and the desired quality level.

A processing function of selected and unselected image frame signalsperformed by the ISP in step 12 of FIG. 1 may be, e.g., processing thecaptured raw image frame data signal into a proper color space format,e.g., RGB (red-green-blue) color space format or a color space formathaving 3 color components, wherein one color component is a luminancecolor component and two other color components are chrominance colorcomponents (e.g., YUV format).

In a next step 14, the selected image frames may be scaled up to beequal in size to the unselected image frames using, e.g., an imagescalar (for displaying the image or for further processing) as shown inFIG. 4 (see modules 50 and 56) and as known in the art: the up-scalingmay be performed, e.g., by averaging of or copying adjacent pixel valuesto added pixels. After step 14, the process may go to step 20 (forfurther processing/encoding/storing as further discussed herein) or tostep 16.

In a next step 16, the selected image frames may be buffered, i.e.,delayed by a predetermined time interval relative to the unselectedimage frames using, e.g., a frame delay controller (e.g., shown asmodule 48 in FIG. 4) in order to avoid jumpy video image displayed on adisplay/viewfinder (i.e., to get the frame timing to match in real timeby maintaining a constant display frame rate as illustrated in FIGS. 2b-2 c and 3 b-3 c described herein). In a next step 18, an imagecomprising interleaving selected and unselected frames with a fasterframe-rate may be displayed (e.g., on a viewfinder of a camera).

In step 20 of FIG. 1, a further processing, e.g., encoding the selectedand unselected frames may be performed with subsequent storing orsending the encoded plurality of frames to a desired destination forpossible displaying. For example, in step 20, a video encoder (forexample, see module 58 in FIG. 4) may encode the YUV frames into acompressed video format file, i.e., MP4/3GPP (Moving Picture ExpertGroup-4/3d Generation Partnership Project) video format. The compressionmay be basically done by generating a high quality video base frameknown as intra-frame (or unselected frame) which is followed by one ormore changed delta video frames (or selected frames) processed andcorrected according to various embodiments of the present invention.This process may be communicated to the FSC (shown as module 54 in FIG.4) via a sync signal 45 which in turn, synchronizes the ISP output frameresolution, so that user requested image size is produced for theintra-frames to be encoded, as disclosed herein. Alternatively, the FSCcan provide the sync signal to the encoder.

Steps 10, 12, 16 and 18 of FIG. 1 may be illustrated using timingdiagrams shown in FIGS. 2 a-2 c and 2 a-3 c as examples of manyscenarios which may be used as described herein.

FIGS. 2 a-2 c show an example among others of timing diagramsdemonstrating alternating one full size unselected image frame 22 (orone reference size frame corresponding to a user requested video imagesize) with one reduced size selected image frame 24, according to anembodiment of the present invention.

FIG. 2 a shows raw image data frames from a sensor as time dependentframe signals 22 and 24 separated by 66 milliseconds (ms) whichcorresponds to the frame rate of about 15 frames per second (fps). FIG.2 b shows a time diagram of selected frames 24 a corresponding to theQQVGA image frame size (160×120 pixels) and unselected frames 22 acorresponding to the QVGA image frame size (320×240 pixels),respectively, after processing by the ISP (per step 12 in FIG. 1),wherein the processing time for the selected frames 24 a is about 32 msshorter than for the unselected frames 22 a. Finally, FIG. 3 c showsimage frame signals (per step 16 in FIG. 1) of unselected and selectedimage frames 22 b and 24 b which are all separated by 66 ms as raw dataimage frames which corresponds to the frame rate of about 15 fps,respectively, such that after the time delay compensation and sizecorrection described herein, the interleaved unselected and selectedimage frames 22 b and 24 b may be displayed on a display/viewfinder (perstep 18 in FIG. 1).

FIGS. 3 a-3 c show another example among others of timing diagramsdemonstrating alternating one full size unselected image frame 26 (orone reference size frame corresponding to a user requested video imagesize) with three reduced size selected image frames 28, according to anembodiment of the present invention.

FIG. 3 a shows raw image data frames from a sensor with time framesignals 26 and 28 separated by 66 milliseconds (ms) which corresponds tothe frame rate of about 15 frames per second (fps). FIG. 3 b shows atime diagram for selected frames 28 a corresponding to the QQVGA imageframe size (160×120 pixels) and unselected frames 26 a corresponding tothe QVGA image frame size (320×240 pixels), respectively, afterprocessing by the ISP (per step 12 in FIG. 1), wherein the processingtime for the selected frames 24 a is about 65 ms shorter than for theselected frames 22 a. Finally, FIG. 3 c shows image frame signals (perstep 16 in FIG. 1) of unselected and selected image frames 26 b and 28 bwhich are all separated by 66 ms as raw data image frames whichcorresponds to the frame rate of about 15 fps, respectively, such thatafter the time delay compensation and size correction described herein,the interleaved unselected and selected image frames 26 b and 28 b maybe displayed on a display/viewfinder (per step 18 in FIG. 1).

FIG. 4 shows an example among others of a block diagram of an electronicdevice 40 (e.g., a camera, a camera phone for wireless applications,etc.) for improving a frame rate by reducing a size/resolution ofselected image frames in an image processing pipeline before displaying(or before encoding stage) as described herein (e.g. see FIG. 1),according to an embodiment of the present invention.

The electronic device 40 may comprise an image sensor 44 (e.g., the CMOSsensor, the CCD sensor or other applicable sensors), for capturing theimage (a lens optics for collecting the image on a sensor surface is notshown in FIG. 4) to provide a raw image data signal (e.g., raw Bayerimage signal) to an interleaving processing unit (or processor) 42 and,in particular, to an image signal processor (ISP) 44 (e.g., a softwareISP). The processor 42 may be a dedicated module or it may beincorporated within other processing modules of the electronic device40. The processor 42 may further comprise a frame synchronizationcontroller (FSC) 54, a frame delay controller 48 and a display imagescalar 50.

The FSC 54 may be used for controlling the output size/resolution of theprocessed video frames and synchronize the scaling operations to be doneon the frames before displaying and/or encoding, as described herein(e.g., see a flow chart of FIG. 1). The ISP 46 may provide processing ofselected images frames (inter-frames) with a reduced size/dimension andunselected image frames (intra-frames), as described herein in regard tostep 12 of FIG. 1, by using an instruction given by the framesynchronization controller (FSC) 54 via the frame size signal 41. TheISP 46 may inform the FSC 54 when the instructed frame is produced viaprocessed frame signal 43. Then the ISP 46 may provide two processedframe signals, a display frame size signal 62 (corresponding to theunselected image frames) and a reduced frame size signal 64(corresponding to the selected image frames).

The frame delay controller 48 may provide a delay of the selected imageframes (e.g., using buffering) by a predetermined time interval relativeto the unselected image frames in order to get the frame timing to matchin real time by maintaining a constant display frame rate, as describedherein in regard to step 16 of FIG. 1. The display image scalar 50 maybe used for scaling up the selected image frames to be equal in size tothe unselected image frames as described herein in regard to step 14 ofFIG. 1, e.g., by averaging of or copying adjacent pixel values to addedpixels. It is noted that the scalar 50 in principle may scale up (ordown) the unselected image frames in a similar manner as discussedherein for the selected image frames to comply with user requested framesize for the unselected frames. It is further noted that according to afurther embodiment, the order for performing steps 14 and 16 of FIG. 1implemented using the corresponding modules 50 and 48 of FIG. 4 may beinterchanged (because scaling operation may be fast and would not have apractical impact on the delay of the up-scaled image frames.

The display image scalar 50 may provide an input signal 68 to a display(e.g., a viewfinder) 52 such as a liquid crystal display (LCD) in orderto display (e.g., in real time) the image comprising interleavedselected and unselected frames with a faster frame-rate, as describedherein in regard to steps 16 and 18 of FIG. 1. After step 14, theprocess may go to step 20 (for further processing/encoding/storing asfurther discussed herein) or to step 16.

However, according to another embodiment, the display image scalar 50may also provide the signal 68 to the encoder 58 for further processing,e.g., encoding the selected and unselected image frames with subsequentstoring in a memory 60 or sending the encoded plurality of image framesto a desired destination using input/output (I/O) port for possibledisplaying/storing/still further processing.

The encoding by the encoder 58 may be performed as described herein inregard to step 20 of FIG. 1. For example, the video encoder 58 mayencode the YUV frames into a compressed video format file such asMP4/3GPP (moving picture expert group-4/3d generation partnershipproject) video format as described herein. This process may becommunicated to the FSC 54 via a sync signal 45 which in turn, maysynchronize the selection of the frame size/resolution, so that userrequested image size may be produced for the intra-frames to be encoded,as disclosed herein. In an alternative implementation, the FSC 54 canprovide the sync signal 45 to the encoder 58.

However, if a color space format (e.g., YUV format) requested forencoding and storing (and/or sending to a desired destination) of thevideo image signal may be different from a color space format (e.g., RGBor black and white) of the video image signal requested, e.g., for thereal-time displaying (on the display 52), then the signal for encodingprovided to the encoder 58 may be processed differently, i.e., withoutusing the signal 68 as an input.

Then if that is the case, according to a further embodiment of thepresent invention, the image signal processor 46 (alternatively, it maybe another dedicated processor) may provide processing of the selectedimages frames (inter-frames) with a reduced size/dimension andunselected image frames (intra-frames), as described herein in regard tostep 12 of FIG. 1, and provide two processed image frame signals, a userrequested frame size signal 73 (corresponding to the unselected imageframes) and user requested reduced frame size signal 74 (correspondingto the selected image frames). Apparently, in this scenario, the signals73 and 74 may be generated by the ISP 46 for a different color spaceformat than the signals 62 and 64. However, it is noted that conversionfrom e.g., the RGB to YUV (or from the YUV to the RGB) color spaceformat may be a standard fast conversion which would not require much ofSW power (alternatively this conversion may be performed as a part ofencoding procedure or just before the encoding by an additionalprocessor). The operation of the image scalar for reduced frame image 56is similar to the operation of the display image scalar 50, as describedherein.

As explained above, the invention provides both a method andcorresponding equipment consisting of various modules providing thefunctionality for performing the steps of the method. The modules may beimplemented as hardware, or may be implemented as software or firmwarefor execution by a computer processor. In particular, in the case offirmware or software, the invention may be provided as a computerprogram product comprising a computer readable storage structure (or acomputer readable medium) embodying a computer program code (i.e., thesoftware or firmware) thereon for execution by the computer processor.

It is noted that various embodiments of the present invention recitedherein may be used separately, combined or selectively combined forspecific applications.

It is to be understood that the above-described arrangements are onlyillustrative of the application of the principles of the presentinvention. Numerous modifications and alternative arrangements may bedevised by those skilled in the art without departing from the scope ofthe present invention, and the appended claims are intended to coversuch modifications and arrangements.

1. A method comprising: processing a plurality of image frames of animage, wherein said processing comprises reducing a size of selectedimage frames chosen from said plurality of image frames relative to anoriginal or a reduced size of unselected image frames of said pluralityof image frames, wherein a processing time for said selected imageframes is shorter than for the unselected image frames; and correctingsaid selected image frames after said processing for displaying saidimage with a targeted frame rate and a desired quality level usinginterleaving of said selected and unselected image frames after saidcorrecting or after said correcting followed by further processing ofsaid selected and unselected image frames.
 2. The method of claim 1,wherein said reducing of the size of the selected image frames isproportional to reducing a resolution of said selected image frames,and, if the original size of the unselected image frames is reduced,said reduced size of the unselected image frames is reduced in the sameproportion as a resolution of the unselected image frames.
 3. The methodof claim 1, wherein said correcting comprises: scaling up said selectedimage frames to be equal in size to said unselected image frames using apre-selected method; and delaying said selected image frames by apredetermined time interval relative to said unselected image frames tomaintain a constant frame rate to provide said displaying of said imagewith the targeted frame rate and the desired quality level.
 4. Themethod of claim 1, wherein said correcting after said processingcomprises scaling up said selected image frames to be equal in size tosaid unselected image frames using a pre-selected method and saidfurther processing comprises encoding of said selected and unselectedimage frames with subsequent storing or sending said selected andunselected frames to a desired destination for said displaying of saidimage with the targeted frame rate and the desired quality level.
 5. Themethod of claim 1, wherein said processing comprises processing intoproper color images which comprises at least one of: a Red-Green-Bluecolor space format or a color space format having three colorcomponents, wherein one color component is a luminance color componentand two other color components are chrominance color components.
 6. Themethod of claim 1, wherein a number of the selected image frames and anumber of the unselected image frames is determined automatically orinputted through a user interface.
 7. The method of claim 1, wherein anumber of the selected image frames and a number of the unselected imageframes automatically vary depending on a moving speed of one or moreobjects in said image in order to provide the targeted frame rate andthe desired quality level.
 8. A computer program product comprising: acomputer readable medium embodying a computer program code thereon forexecution by a computer processor with said computer program code,wherein said computer program code comprises instructions for performingthe method of claim
 1. 9. An apparatus, comprising: an image signalprocessor, configured to perform processing of a plurality of imageframes of an image, said processing comprises reducing a size ofselected image frames chosen from said plurality of image framesrelative to an original or a reduced size unselected image frames ofsaid plurality of image frames, wherein a processing time for saidselected image frames is shorter than for the unselected image frames;and a frame synchronization controller, configured to provide aselection signal to choose said selected image frames, wherein saidapparatus is configured to provide correcting said selected image framesafter said processing in order to display said image with a targetedframe rate and a desired quality level using interleaving of saidselected and unselected image frames after said correcting or after saidcorrecting followed by further processing of said selected andunselected image frames.
 10. The apparatus of claim 9, furthercomprising: a display image scalar, configured to scale up, afterperforming said processing, said selected image frames to be equal insize to said unselected image frames using a pre-selected method as apart of said correcting.
 11. The apparatus of claim 9, furthercomprising: a delay unit, configured to delay said selected imageframes, after performing said processing, by a predetermined timeinterval relative to said unselected image frames to maintain a constantframe rate as a part of said correcting to provide said displaying ofsaid image.
 12. The apparatus of claim 9, further comprising: anencoder, configured to perform said further processing by encoding saidselected and unselected image frames which is followed by storing orsending said selected and unselected frames to a desired destination forsaid displaying of said image.
 13. The apparatus of claim 9, whereinsaid plurality of image frames is raw image data and the apparatuscomprises a sensor configured to provide said raw image data signal. 14.The apparatus of claim 13, wherein said sensor is a complimentary metaloxide semiconductor or a charged-couple device.
 15. The apparatus ofclaim 9, wherein said image signal processor and said framesynchronization controller are combined in one unit or a processor. 16.The apparatus of claim 9, wherein a camera or an electronic devicecomprising a camera comprises said image signal processor and said framesynchronization controller.
 17. The apparatus of claim 9, wherein,during said processing, said selected image frames are reduced to a sizeof 160×120 pixels, and said unselected image frames of the plurality ofimage frames are reduced to a size of 320×240 pixels.
 18. The apparatusof claim 9, wherein each of said selected image frames is chosen asbeing every second frame from said plurality of image frames.
 19. Theapparatus of claim 9, wherein the apparatus is configured to determine,using a processor performance measure of the image signal processor inorder to achieve the targeted frame rate and the desired quality levelat least one of: a number of the selected image frames and a number ofthe unselected image frames, and a size of the selected image frames anda size of the unselected image frames.
 20. A processor, comprising: animage signal processor, configured to perform processing of a pluralityof image frames of an image, said processing comprises reducing a sizeof selected image frames chosen from said plurality of image framesrelative to an original or a reduced size of unselected image frames ofsaid plurality of image frames, wherein a processing time for saidselected image frames is shorter than for the unselected image frames;and a frame synchronization controller, configured to provide aselection signal to choose said selected image frames, wherein saidapparatus is configured to provide correcting said selected image framesafter said processing in order to display said image with a targetedframe rate and a desired quality level using interleaving of saidselected and unselected image frames after said correcting or after saidcorrecting followed by further processing of said selected andunselected image frames.